1. Field of the Invention
The present invention relates to an image processing system in which an image processing device and an image processing terminal are connected to each other via a network, and also relates to an image processing device and an image processing terminal used in the image processing system.
2. Description of the Related Art
One of physical quantities expressing a physical property specific to a subject of imaging is a spectral transmittance spectrum. Spectral transmittance is a physical quantity expressing a ratio of transmitted light to incident light at each wavelength, and is specific information of an object, whose value does not change due to an extrinsic influence. It is different from color information that depends on a change of illumination light, such as an RGB value. Therefore, the spectral transmittance is used in various fields as information for reproducing the color of the subject itself. For example, for a body tissue sample, particularly in the field of pathological diagnosis using pathological specimens, spectral transmittance has been used as an example of a spectral characteristic value for analysis of images acquired by imaging samples.
In pathological diagnosis, a process is widely practiced such that a pathological specimen is magnified to be observed using a microscope after slicing a block sample obtained by excision of an organ or a pathological specimen obtained by needle biopsy into pieces having a thickness of about several microns to obtain various findings. Transmission observation using an optical microscope is one of observation methods most widely practiced, because materials for optical microscopes are relatively inexpensive and easy to handle, and this method has been used for many years. In a case of transmission observation, because a sliced sample hardly absorbs or scatters light and is substantially transparent and colorless, it is common to stain the sample with a dye prior to observation.
Various methods have been proposed as the staining method, and there have been more than a hundred methods in total. Particularly for pathological specimens, hematoxylin-eosin stain (hereinafter, “H&E stain”) using bluish purple hematoxylin and red eosin has been generally used.
Hematoxylin is a natural substance extracted from plants, and has no stainability itself. However, hematin, which is an oxide of hematoxylin, is a basophilic dye and combines with a substance negatively charged. Because deoxyribonucleic acid (DNA) included in a cell nucleus is negatively charged due to a phosphate group included therein as a structural element, the DNA combines with hematin to be stained bluish purple. As described above, substance having stainability is not hematoxylin but its oxide, namely hematin. However, because it is common to use hematoxylin as the name of dye, this applies to the following explanations. Meanwhile, eosin is an acidophilic dye, and combines with a substance positively charged. Amino acid and protein are charged positively or negatively depending on its pH environment, and have a strong tendency to be charged positively under acidity. For this reason, there are cases that acetic acid is added to eosin. The protein included in a cytoplasm combines with eosin to be stained red or light red.
In a sample subjected to H&E stain (a stained sample), cell nucleuses, bone tissues or the like are stained bluish purple, and cytoplasm, connective tissues, red corpuscles or the like are stained red, to have them become easily visible. Accordingly, an observer can ascertain the size, positional relation or the like of elements structuring a cell nuclei or the like, thereby enabling to determine a state of the sample morphologically.
Observation of samples is performed by multiband imaging the samples to be displayed on a display screen of an external device, in addition to visual inspection by an observer. In a case of displaying images on a display screen, processing for estimating spectral transmittance at each sample point from the obtained multiband image, processing for estimating a dye amount of a dye with which the sample is stained based on the estimated spectral transmittance, processing for correcting the color of the image based on the estimated dye amount or the like are performed. Variation in the property of camera, the stained state or the like are then corrected, and an RGB image for display of the samples is composed. FIG. 38 is an example of a composed display image. When the estimation of a dye amount is appropriately performed, samples stained darker or stained lighter can be corrected to an image in a color equivalent to a sample that is properly stained.
As a method of estimating spectral transmittance at each sample point from a multiband image of samples, for example, an estimation method by principal component analysis (see, for example, “Development of support systems for pathology using spectral transmittance—The quantification method of stain conditions”, Proceedings of SPIE, Vol. 4684, 2003, p. 1516 to 1523), and an estimation method by the Wiener estimation (for example, see “Color Correction of Pathological Images Based on Dye Amount Quantification”, OPTICAL REVIEW, Vol. 12, No. 4, 2005, p. 293-300) can be mentioned. The Wiener estimation is widely known as a technique of linear filtering methods for estimating an original signal from an observed signal on which noise is superimposed, which is a method for minimizing an error, by taking into consideration statistical properties of an observed object and properties of imaging noise (observation noise). Because some noise is included in signals from a camera, the Wiener estimation is very useful as a method for estimating an original signal.
A method of combining display images from multiband images of a sample is explained below. First, a multiband image of a sample is captured. For example, a technique disclosed in Japanese Laid-open Patent Publication No. 07-120324 is used to capture a multiband image according to a frame sequential method, while switching 16 pieces of bandpass filters by rotating a filter wheel. In this way, multiband images having a pixel value of 16 bands at each point of the sample can be obtained. Although the dye is three-dimensionally distributed in the sample as the original observed object, it cannot be captured as a three-dimensional image as it is with an ordinary transmission observation system, and is observed as a two-dimensional image in which illumination light that has passed the sample is projected on an imaging element of the camera. Accordingly, each point mentioned herein signifies a point on the sample corresponding to each projected pixel of the imaging element.
For an arbitrary point (pixel) x of a captured multiband image, a relation expressed by the following Equation (1) based on a response system of the camera is established between a pixel value g(x,b) in a band b and spectral transmittance t(x,λ) of a corresponding point on the sample.g(x,b)=∫λf(b,λ)s(λ)e(λ)t(x,λ)dλ+n(b)  (1)
In Equation (1), λ denotes a wavelength, f(b,λ) denotes a spectral-transmittance of a bth filter, s(λ) denotes a spectral sensitivity characteristic of the camera, e(λ) denotes a spectral emission characteristic of illumination, and n(b) denotes imaging noise in the band b. B denotes a serial number for identifying the band, and is an integer satisfying 1≦b≦16.
In practical calculation, the following Equation (2) obtained by the discretizing Equation (1) in a wavelength direction is used.G(x)=FSET(x)+N  (2)
When the number of samples in the wavelength direction is designated as D, and the number of bands is designated as B (here, B=16), G(x) denotes a matrix of B rows by one column corresponding to a pixel value g(x,b) at a point x. Similarly, T(x) denotes a matrix of D rows by one column corresponding to t(x,λ), and F denotes a matrix of B rows by D columns corresponding to f(b,λ). On the other hand, S denotes a diagonal matrix of D rows by D columns, and a diagonal element corresponds to s(λ). Similarly, E denotes a diagonal matrix of D rows by D columns, and a diagonal element corresponds to e(λ). N denotes a matrix of B rows by one column corresponding to n(b). In Equation (2), because expressions of a plurality of bands are put together using a matrix, a variable b expressing the band is not explicitly described. Further, an integral of a wavelength λ is replaced by a product of matrices.
To simplify description, a matrix H defined by the following Equation (3) is introduced. H is also called as a system matrix.H=FSE  (3)
Thus, Equation (3) is replaced by the following Equation (4)G(x)=HT(x)+N  (4)
The spectral transmittance at each sample point is then estimated from the captured multiband image by using the Wiener estimation. An estimate value (spectral transmittance data) {circumflex over (T)}(x) of the spectral transmittance can be calculated by the following Equation (5).{circumflex over (T)}(x)=WG(x)  (5)
W is expressed by the following Equation (6), and is referred to as “Wiener estimation matrix” or “estimation operator used in the Wiener estimation”.W=RSSHt(HRSSHt+RNN)−1  (6)where ( )t: transposed matrix, ( )−1: inverse matrix.
In Equation (5), RSS is a matrix of D rows by D columns and represents an autocorrelation matrix of the spectral transmittance of the sample. RNN is a matrix of B rows by B columns and represents an autocorrelation matrix of noise of the camera used for imaging.
After thus estimating spectral transmittance data {circumflex over (T)}(x), amounts of dyes at a corresponding point on the sample (a sample point) are estimated based on the {circumflex over (T)}(x). Dyes to be estimated are three kinds of dyes, which are hematoxylin, eosin that stains a cell cytoplasm, and eosin that stains red blood cells or an original dye of the red blood cells that are not stained. These three kinds of dyes are abbreviated as a dye H, a dye E, and a dye R, respectively. To be strict, the red blood cells have a specific color itself even in an unstained state, and after the H&E stain is performed, the color of the red blood cells and the color of eosin that has changed in a staining process are superposed on each other at the time of observation. Therefore, in precise, color obtained by combining the both is referred to as the dye R.
Generally, in a substance that transmits light, it is known that the Lambert-Beer law represented by the following Equation (7) is established between an intensity I0(λ) of incident light and an intensity I(λ) of emitted light at each wavelength λ.
                                          I            ⁡                          (              λ              )                                                          I              0                        ⁡                          (              λ              )                                      =                  ⅇ                                    -                              k                ⁡                                  (                  λ                  )                                                      ·            d                                              (        7        )            
In Equation (7), k(λ) denotes a value specific to a substance determined depending on the wavelength, and d denotes a thickness of the substance.
The left side of Equation (7) indicates a spectral transmittance t(λ), and Equation (7) is replaced by the following Equation (8).t(λ)=e−k(λ)·d  (8)
Further, a spectral absorbance a(λ) is represented by the following Equation (9).a(λ)=k(λ)·d  (9)
Thus, Equation (8) is replaced by the following Equation (10).t(λ)=e−a(λ)  (10)
When an H&E stained sample is stained with three kinds of dyes of the dye H, the dye E, and the dye R, the following Equation (11) is established at each wavelength λ by the Lambert-Beer law.
                                          I            ⁡                          (              λ              )                                                          I              0                        ⁡                          (              λ              )                                      =                  ⅇ                      -                          (                                                                                          k                      H                                        ⁡                                          (                      λ                      )                                                        ·                                      d                    H                                                  +                                                                            k                      E                                        ⁡                                          (                      λ                      )                                                        ·                                      d                    E                                                  +                                                                            k                      R                                        ⁡                                          (                      λ                      )                                                        ·                                      d                    R                                                              )                                                          (        11        )            where kH(λ), kE(λ), and kR(λ) denote k(λ) corresponding to the dye H, the dye E, and the dye R, respectively, and for example, are dye spectra of respective dyes that stain the sample (hereinafter, “reference dye spectra”). Further, dH, dE, and dR indicate a virtual thickness of the dye H, the dye E, and the dye R at each sample point corresponding to each image position of the multiband image. Basically, dyes are dispersed in a sample, and thus the thickness is not a correct idea. However, this can be an index of a relative dye amount that indicates how much amount of dye is present, as compared to a case that the sample is assumed to be stained with a single dye. That is, it can be said that dH, dE, and dR indicate a dye amount of the dye H, the dye E, and the dye R, respectively. The values kH(λ), kE(λ), and kR(λ) can be easily acquired from the Lambert-Beer law, by preparing samples that are stained individually by using the dye H, the dye E, and the dye R, and measuring a spectral transmittance thereof with a spectrometer.
When it is assumed that a spectral transmittance at a position x is t(x,λ) and a spectral absorbance at the position x is a(x,λ), Equation (9) can be replaced by the following Equation (12).a(x,λ)=kH(λ)·dH+kE(λ)·dE+kR(λ)·dR  (12)
When it is assumed that an estimated spectral transmittance at the wavelength λ of the spectral transmittance {circumflex over (T)}(x) estimated by using Equation (5) is {circumflex over (t)}(x,λ), and an estimated absorbance is â(x,λ), Equation (12) can be replaced by the following Equation (13).{circumflex over (a)}(x,λ)=kH+kE(λ)·dE+kR(λ)·dR  (13)
In Equation (13), unknown variables are three variables of dH, dE, and dR. Therefore, when simultaneous Equations are acquired from Equation (13) for at least three different wavelengths λ, these can be solved. To further improve the accuracy, simultaneous Equations can be acquired from Equation (13) for four or more different wavelengths λ, to perform multiple regression analysis. For example, simultaneous Equations acquired from Equation (13) for three wavelengths λ1, λ2, and λ3 can be expressed in a matrix as the following Equation (14).
                              (                                                                                          a                    ^                                    ⁡                                      (                                          x                      ,                                              λ                        1                                                              )                                                                                                                                            a                    ^                                    ⁡                                      (                                          x                      ,                                              λ                        2                                                              )                                                                                                                                            a                    ^                                    ⁡                                      (                                          x                      ,                                              λ                        3                                                              )                                                                                )                =                              (                                                                                                      k                      H                                        ⁡                                          (                                              λ                        1                                            )                                                                                                                                  k                      E                                        ⁡                                          (                                              λ                        1                                            )                                                                                                                                  k                      R                                        ⁡                                          (                                              λ                        1                                            )                                                                                                                                                              k                      H                                        ⁡                                          (                                              λ                        2                                            )                                                                                                                                  k                      E                                        ⁡                                          (                                              λ                        2                                            )                                                                                                                                  k                      R                                        ⁡                                          (                                              λ                        2                                            )                                                                                                                                                              k                      H                                        ⁡                                          (                                              λ                        3                                            )                                                                                                                                  k                      E                                        ⁡                                          (                                              λ                        3                                            )                                                                                                                                  k                      R                                        ⁡                                          (                                              λ                        3                                            )                                                                                            )                    ⁢                      (                                                                                d                    H                                                                                                                    d                    E                                                                                                                    d                    R                                                                        )                                              (        14        )            
Equation (14) is replaced here by the following Equation (15).{circumflex over (A)}(x)=Kd(x)  (15)
When the number of samples in a wavelength direction is D, Â(x) is a matrix of D rows and one column corresponding to â(x,λ), K is a matrix of D rows and three columns corresponding to k(λ), and d(x) is a matrix of three rows and one column corresponding to dH, dE, and dR at point x.
According to Equation (15), the dye amounts dH, dE, and dR are calculated using a least square method. The least square method is a method of determining d(x) such that a square sum of an error is minimized in a single regression Equation, and the dye amounts can be calculated by the following Equation (16).{circumflex over (d)}(x)=(KTK)−1KTÂ(x)  (16)
In Equation (16), {circumflex over (d)}(x) is an estimated dye amount.
When dye amounts {circumflex over (d)}H, {circumflex over (d)}E, {circumflex over (d)}R are estimated for the dye H, the dye E, and the dye R and substituted in Equation (12), a restored spectral absorbance ã(x,y) can be obtained according to the following Equation (17).{tilde over (a)}(x,λ)=kH(λ)·{circumflex over (d)}H+kE(λ)·{circumflex over (d)}E+kR(λ)·{circumflex over (d)}R  (17)
An estimated error e(λ) in estimation of dye amount is obtained based on the estimated spectral absorbance â(x,λ) and the restored spectral absorbance ã(x,y) according to the following Equation (18). Hereinafter, e(λ) is referred to as “residual spectrum”.e(λ)={circumflex over (a)}(x,λ)−{tilde over (a)}(x,λ)  (18)
The estimated spectral absorbance â(x, λ) can be represented by the following Equation (19) based on Equations (17) and (18).{circumflex over (a)}(x,λ)=kH(λ)·{circumflex over (d)}H+kE(λ)·{circumflex over (d)}E+kR(λ)·{circumflex over (d)}R+e(λ)  (19)
The Lambert-Beer law formulates attenuation of light transmitting through a semi-transparent substance while assuming that there is no refraction or scattering. However, in an actual sample, refraction and scattering can occur. Therefore, when attenuation of light due to the sample is modeled only by the Lambert-Beer law, an error resulting from refraction or scattering occurs. However, it is quite difficult to construct a model including refraction or scattering. Therefore, unnatural color variation by a physical model can be prevented by taking into consideration the residual spectrum e(λ), which is a modeling error including influences of refraction and scattering.
When the dye amounts {circumflex over (d)}H, {circumflex over (d)}E, {circumflex over (d)}R are determined in this manner, a change in the dye amounts in the sample can be simulated by correcting the dye amounts. The dye amounts {circumflex over (d)}H and {circumflex over (d)}E stained by a staining method are corrected here. The dye amount {circumflex over (d)}R, which is an original color of the red blood cell, is not corrected. That is, corrected dye amounts {circumflex over (d)}H*, {circumflex over (d)}E* can be obtained by using appropriate dye-amount correction coefficients αH and αE, according to the following Equations (20) and (21).{circumflex over (d)}H*=αH{circumflex over (d)}H  (20){circumflex over (d)}E*=αE{circumflex over (d)}E  (21)
When the obtained corrected dye amounts {circumflex over (d)}H*, {circumflex over (d)}E* are substituted in Equation (12), a spectral absorbance ã*(x,y) can be obtained according to the following Equation (22).{tilde over (a)}*(x,λ)=kH(λ)·{circumflex over (d)}H*+kE(λ)·{circumflex over (d)}E*+kR(λ)·{circumflex over (d)}R  (22)
Further, when the residual spectrum e(λ) is included, a new spectral absorbance â*(x, λ) can be obtained according to Equation (23).{circumflex over (a)}*(x,λ)=kH(λ)·{circumflex over (d)}H*kE(λ)·{circumflex over (d)}E*kR(λ)·{circumflex over (d)}R+e(λ)  (23)
When the spectral absorbance ã*(x,y) or spectral absorbance â*(x, λ) is substituted in Equation (10), a new spectral transmittance t*(x,λ) is obtained according to Equation (24). A spectral absorbance a*(x,λ) is a value of the spectral absorbance ã*(x,y) or spectral absorbance â*(x, λ).t*(x,λ)=e−a*(x,λ)  (24)
When Equation (24) is substituted in Equation (1), a new pixel value g*(x,b) can be obtained by the following Equation (25). In this case, it can be calculated while assuming that observation noise n(b) is zero.g*(x,b)=∫λf(b,λ)s(λ)e(λ)t*(x,λ)dλ  (25)
Equation (4) is replaced here by the following Equation (26).G*(x)=HT*(x)  (26)
G*(x) is a matrix of B rows and one column corresponding to g*(x,b), and T*(x) is a matrix of D rows and one column corresponding to t*(x,λ). Accordingly, a pixel value G*(x) of the sample in which the dye amount is virtually changed can be synthesized.
As explained above, by estimating the dye amounts at an arbitrary point x in a multiband image by the procedure described above to virtually adjust the dye amounts at each sample point and synthesizing an image of the sample after adjustment, the dye amounts of the sample can be corrected. At this time, the dye amounts at each sample point can be adjusted to an appropriate stained state by automatically performing color normalization. Further, when an appropriate user interface is prepared, a user can adjust the dye amounts manually. A display image synthesized for display is, for example, screen-displayed on a display device, and used for pathological diagnosis by a doctor or the like. Accordingly, even if there is a variation in stain of the sample, an image that is adjusted to an appropriate stained state can be observed.